Factors affecting visual influence on heard vowel roun-dedness: Web experiments with Swedes and Turks

The influence of various general and stimulusspecific factors on the contribution of vision to heard roundedness was investigated by means of web experiments conducted in Swedish. Introduction The original utterances consisted of the syllables /ɡyːɡ/ and /ɡeːɡ/ of a male and a female speaker. They were synchronized with each other in all combinations, resulting in four stimuli that were incongruent in vowel quality, two of them additionally in speaker sex. One of the experiments was also conducted in Turkish, using the same stimuli. The results showed that visible presence of lip rounding has a weaker effect on audition than its absence, except for conditions that evoke increased attention, such as when a foreign language is involved. The results suggest that female listeners are more susceptible to vision under such conditions. There was no significant effect of age and of discomfort felt by being exposed to dubbed speech. A discrepancy in speaker sex did not lead to reduced influence of vision. The results also showed that habituation to dubbed speech has no deteriorating effect on normal auditory-visual integration in the case of roundedness. In auditory speech perception, the perceptual weight of the information conveyed by the visible face of a speaker can be expected to vary with many factors. 1) The particular phonetic feature and system 2) Language familiarity 4) The individual speaker and speech style 3) The individual perceiver 5) Visibility of the face / audibility of the voice 6) The perceiver’s knowledge about the stimuli 7) Context 8) Cultural factors Most studies within this field have been concerned with the perception of place of articulation in consonants, like McGurk and MacDonald (1976). These studies have shown that the presence/absence of labial closure tends to be perceived by vision. As for vowels, it is known that under ideal audibility and visibility conditions, roundedness is largely heard by vision, while heard openness (vowel height) is hardly at all influenced by vision (Traunmüller & Öhrström, 2007). These observations make it clear that the presence/absence of features tends to be perceived by vision if their auditory cues are subtle while their visual cues are prominent. Differences between phonetic systems are also relevant. When, e.g. an auditory [ɡ] is presented in synchrony with a visual [b], this is likely to fuse into a [ɡ͡b] only for perceivers who are competent in a language with a [ɡ͡b]. Others are more likely to perceive a [ɡ] or a consonant cluster. The observed lower visual influence in speakers of Japanese as compared with English (Sekiyama and Burnham, 2008) represents a more subtle case, whose cause may lie outside the phonetic system. The influence of vision is increased when the perceived speech sounds foreign (Sekiyama and Tohkura, 1993; Hayashi and Sekiyama, 1998; Chen and Hazan, 2007). This is referred to as the “foreign-language effect”. The influence of vision varies substantially between speakers and speaking styles (Munhall et al., 1996; Traunmüller and Öhrström, 2007). The influence of vision also varies greatly between perceivers. There is variation with age. Pre-school children are less sensitive (Sekiyama and Burnham, 2004) although even prelinguistic children show influence of vision (Burnham and Dodd, 2004). There is also a subtle sex difference: Women tend to be more susceptible to vision (Irwin et al., 2006; Traunmüller and Öhrström, 2007). The influence of vision increases with decreasing audibility of the voice, e.g. due to noise, and decreases with decreasing visibility of the face, but only very litte with increasing distance up to 10 m (Jordan, 2000). Auditory-visual integration works even when there is a discrepancy in sex between a voice and a synchronized face (Green et al., 1991) and it is also robust with respect to what the perceiver is told about the stimuli. A minor effect on vowel perception has, nevertheless, been observed when subjects were told the sex Proceedings, FONETIK 2009, Dept. of Linguistics, Stockholm University represented by an androgynous voice (Johnson, Strand and d’Imperio, 1991). Auditory-visual integration is robust to semantic factors (Sams et al, 1998) but it is affected by context, e.g. the vocalic context of consonants (Shigeno, 2002). It can also be affected by the experimental method (e.g., blocked vs. random stimulus presentation). It has been suggested that cultural conventions, such as socially prescribed gaze avoidance, may affect the influence of vision (Sekiyama and Tohkura, 1993; Sekiyama, 1997). Exposure to dubbed films is another cultural factor that can be suspected to affect the influence of vision. The dubbing of foreign movies is a widespread practice that often affects nearly all speakers of certain languages. Since in dubbed speech, the sound is largely incongruent with the image, habituation requires learning to disrupt the normal process of auditory-visual integration. Considering also that persons who are not habituated often complain about discomfort and mental pain when occasionally exposed to dubbed speech, it deserves to be investigated whether the practice of dubbing deteriorates auditory-visual integration more permanently in the exposed populations. The present series of web experiments had the primary aim of investigating (1) the effects of the perceiver’s knowledge about the stimuli and (2) those of a discrepancy between face and voice (male/female) on the heard presence or absence of liprounding in front vowels. Additional factors considered, without being experimentally balanced, were (3) sex and (4) age of the perceiver, (5) discomfort felt from dubbed speech, (6) noticed/unnoticed phonetic incongruence and (7) listening via loudspeaker or headphones. The experiments were conducted in Swedish, but one experiment was also conducted in Turkish. The language factor that may disclose itself in this way has to be interpreted with caution, since (8) the “foreign-language effect” remains confounded with (9) effects due to the difference between the phonetic systems. Method Most Turks are habituated to dubbed speech, since dubbing foreign movies into Turkish is fairly common. Some are not habituated, since such dubbing is not pervasive. This allows investigating (10) the effect of habituation to dubbed speech. Since dubbing into Swedish is only rarely practiced with performances intended for children adult Swedes are rarely habituated to dubbed speech. Speakers The speakers were two native Swedes, a male doctoral student, 29 years (index ♂), and a female student, 21 years (index ♀). These were two of the four speakers who served for the experiments reported in Traunmüller and Öhrström (2007). For the present experiment, a selection of audiovisual stimuli from this experiment was reused.